{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec\n",
    "\n",
    "原理：https://www.jianshu.com/p/471d9bfbd72f\n",
    "\n",
    "数学基础：https://www.cnblogs.com/peghoty/p/3857839.html\n",
    "\n",
    "> - 如果是用一个词语作为输入，来预测它周围的上下文，那这个模型叫做 __Skip-gram 模型__\n",
    "> - 而如果是拿一个词语的上下文作为输入，来预测这个词语本身，则是 __CBOW 模型__\n",
    "\n",
    "## Skip-gram和CBOW的通用形式:\n",
    "__用当前词 x 预测它的下一个词 y__\n",
    "\n",
    "> - input: 输入词的one-hot encoder\n",
    "> - output: 输出词的one-hot encoder\n",
    "\n",
    "![image](s.png)\n",
    "\n",
    "\n",
    "## Skip-gram\n",
    "__当 y 有多个词时__\n",
    "> - input: 输入词的one-hot encoder\n",
    "> - multi-output: 输出词的one-hot encoder\n",
    "\n",
    "![image](skip-gram.png)\n",
    "\n",
    "## CBOW\n",
    "__当 x 有多个词时__\n",
    "> - multi-input: 输入词的one-hot encoder\n",
    "> - output: 输出词的one-hot encoder\n",
    "\n",
    "![image](CBOW.png)\n",
    "\n",
    "## Hierarchical Softmax\n",
    "__层次softmax的核心内容是哈夫曼树（Huffman Tree），树的核心概念是 出现概率越高的符号使用较短的编码（层次越浅），出现概率低的符号则使用较长的编码（层次越深)。__\n",
    "\n",
    "![image](Hierarchical_softmax.png)\n",
    "\n",
    "## 负采样\n",
    "__任何采样算法都应该保证频次越高的样本越容易被采样出来。__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x10caf7950>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see http://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html\n",
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vocab_size: 49\n"
     ]
    }
   ],
   "source": [
    "CONTEXT_SIZE = 2  # 2 words to the left, 2 to the right\n",
    "text = \"\"\"We are about to study the idea of a computational process.\n",
    "Computational processes are abstract beings that inhabit computers.\n",
    "As they evolve, processes manipulate other abstract things called data.\n",
    "The evolution of a process is directed by a pattern of rules\n",
    "called a program. People create programs to direct processes. In effect,\n",
    "we conjure the spirits of the computer with our spells.\"\"\".split()\n",
    "\n",
    "split_ind = (int)(len(text) * 0.8)\n",
    "\n",
    "# By deriving a set from `raw_text`, we deduplicate the array\n",
    "vocab = set(text)\n",
    "vocab_size = len(vocab)\n",
    "print('vocab_size:', vocab_size)\n",
    "\n",
    "w2i = {w: i for i, w in enumerate(vocab)}\n",
    "i2w = {i: w for i, w in enumerate(vocab)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'about': 0, 'of': 1, 'beings': 2, 'rules': 3, 'computers.': 4, 'In': 5, 'effect,': 6, 'programs': 7, 'Computational': 8, 'computational': 9, 'data.': 10, 'to': 11, 'conjure': 12, 'We': 13, 'spirits': 14, 'things': 15, 'computer': 16, 'is': 17, 'manipulate': 18, 'with': 19, 'People': 20, 'evolve,': 21, 'study': 22, 'other': 23, 'idea': 24, 'process.': 25, 'evolution': 26, 'directed': 27, 'pattern': 28, 'processes': 29, 'spells.': 30, 'program.': 31, 'inhabit': 32, 'we': 33, 'by': 34, 'our': 35, 'abstract': 36, 'create': 37, 'they': 38, 'the': 39, 'As': 40, 'a': 41, 'process': 42, 'processes.': 43, 'that': 44, 'The': 45, 'are': 46, 'direct': 47, 'called': 48}\n"
     ]
    }
   ],
   "source": [
    "print(w2i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'about',\n",
       " 1: 'of',\n",
       " 2: 'beings',\n",
       " 3: 'rules',\n",
       " 4: 'computers.',\n",
       " 5: 'In',\n",
       " 6: 'effect,',\n",
       " 7: 'programs',\n",
       " 8: 'Computational',\n",
       " 9: 'computational',\n",
       " 10: 'data.',\n",
       " 11: 'to',\n",
       " 12: 'conjure',\n",
       " 13: 'We',\n",
       " 14: 'spirits',\n",
       " 15: 'things',\n",
       " 16: 'computer',\n",
       " 17: 'is',\n",
       " 18: 'manipulate',\n",
       " 19: 'with',\n",
       " 20: 'People',\n",
       " 21: 'evolve,',\n",
       " 22: 'study',\n",
       " 23: 'other',\n",
       " 24: 'idea',\n",
       " 25: 'process.',\n",
       " 26: 'evolution',\n",
       " 27: 'directed',\n",
       " 28: 'pattern',\n",
       " 29: 'processes',\n",
       " 30: 'spells.',\n",
       " 31: 'program.',\n",
       " 32: 'inhabit',\n",
       " 33: 'we',\n",
       " 34: 'by',\n",
       " 35: 'our',\n",
       " 36: 'abstract',\n",
       " 37: 'create',\n",
       " 38: 'they',\n",
       " 39: 'the',\n",
       " 40: 'As',\n",
       " 41: 'a',\n",
       " 42: 'process',\n",
       " 43: 'processes.',\n",
       " 44: 'that',\n",
       " 45: 'The',\n",
       " 46: 'are',\n",
       " 47: 'direct',\n",
       " 48: 'called'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i2w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cbow sample (['We', 'are', 'to', 'study'], 'about')\n",
      "skipgram sample ('about', 'We', 1)\n"
     ]
    }
   ],
   "source": [
    "# context window size is two\n",
    "def create_cbow_dataset(text):\n",
    "    data = []\n",
    "    for i in range(2, len(text) - 2):\n",
    "        context = [text[i - 2], text[i - 1],\n",
    "                   text[i + 1], text[i + 2]]\n",
    "        target = text[i]\n",
    "        data.append((context, target))\n",
    "    return data\n",
    "\n",
    "def create_skipgram_dataset(text):\n",
    "    import random\n",
    "    data = []\n",
    "    for i in range(2, len(text) - 2):\n",
    "        data.append((text[i], text[i-2], 1))\n",
    "        data.append((text[i], text[i-1], 1))\n",
    "        data.append((text[i], text[i+1], 1))\n",
    "        data.append((text[i], text[i+2], 1))\n",
    "        # negative sampling\n",
    "        for _ in range(4):\n",
    "            if random.random() < 0.5 or i >= len(text) - 3:\n",
    "                rand_id = random.randint(0, i-1)\n",
    "            else:\n",
    "                rand_id = random.randint(i+3, len(text)-1)\n",
    "            data.append((text[i], text[rand_id], 0))\n",
    "    return data\n",
    "\n",
    "cbow_train = create_cbow_dataset(text)\n",
    "skipgram_train = create_skipgram_dataset(text)\n",
    "print('cbow sample', cbow_train[0])\n",
    "print('skipgram sample', skipgram_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOW(nn.Module):\n",
    "    def __init__(self, vocab_size, embd_size, context_size, hidden_size):\n",
    "        super(CBOW, self).__init__()\n",
    "        self.embeddings = nn.Embedding(vocab_size, embd_size)\n",
    "        self.linear1 = nn.Linear(2*context_size*embd_size, hidden_size)\n",
    "        self.linear2 = nn.Linear(hidden_size, vocab_size)\n",
    "        \n",
    "    def forward(self, inputs):\n",
    "        embedded = self.embeddings(inputs).view((1, -1))\n",
    "        hid = F.relu(self.linear1(embedded))\n",
    "        out = self.linear2(hid)\n",
    "        log_probs = F.log_softmax(out)\n",
    "        return log_probs\n",
    "\n",
    "class SkipGram(nn.Module):\n",
    "    def __init__(self, vocab_size, embd_size):\n",
    "        super(SkipGram, self).__init__()\n",
    "        self.embeddings = nn.Embedding(vocab_size, embd_size)\n",
    "    \n",
    "    def forward(self, focus, context):\n",
    "        embed_focus = self.embeddings(focus).view((1, -1))\n",
    "        embed_ctx = self.embeddings(context).view((1, -1))\n",
    "        score = torch.mm(embed_focus, torch.t(embed_ctx))\n",
    "        log_probs = F.logsigmoid(score)\n",
    "    \n",
    "        return log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CBOW(\n",
      "  (embeddings): Embedding(49, 100)\n",
      "  (linear1): Linear(in_features=400, out_features=64, bias=True)\n",
      "  (linear2): Linear(in_features=64, out_features=49, bias=True)\n",
      ")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rayest/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:12: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SkipGram(\n",
      "  (embeddings): Embedding(49, 100)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "embd_size = 100\n",
    "learning_rate = 0.001\n",
    "n_epoch = 30\n",
    "\n",
    "def train_cbow():\n",
    "    hidden_size = 64\n",
    "    losses = []\n",
    "    loss_fn = nn.NLLLoss()\n",
    "    model = CBOW(vocab_size, embd_size, CONTEXT_SIZE, hidden_size)\n",
    "    print(model)\n",
    "    optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
    "\n",
    "    for epoch in range(n_epoch):\n",
    "        total_loss = 0\n",
    "        for context, target in cbow_train:\n",
    "            ctx_idxs = [w2i[w] for w in context]\n",
    "            ctx_var = Variable(torch.LongTensor(ctx_idxs))\n",
    "\n",
    "            model.zero_grad()\n",
    "            log_probs = model(ctx_var)\n",
    "\n",
    "            loss = loss_fn(log_probs, Variable(torch.LongTensor([w2i[target]])))\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            total_loss += loss.data\n",
    "        losses.append(total_loss)\n",
    "    return model, losses\n",
    "\n",
    "def train_skipgram():\n",
    "    losses = []\n",
    "    loss_fn = nn.MSELoss()\n",
    "    model = SkipGram(vocab_size, embd_size)\n",
    "    print(model)\n",
    "    optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
    "    \n",
    "    for epoch in range(n_epoch):\n",
    "        total_loss = .0\n",
    "        for in_w, out_w, target in skipgram_train:\n",
    "            in_w_var = Variable(torch.LongTensor([w2i[in_w]]))\n",
    "            out_w_var = Variable(torch.LongTensor([w2i[out_w]]))\n",
    "            \n",
    "            model.zero_grad()\n",
    "            log_probs = model(in_w_var, out_w_var)\n",
    "            loss = loss_fn(log_probs[0], Variable(torch.Tensor([target])))\n",
    "            \n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            total_loss += loss.data\n",
    "        losses.append(total_loss)\n",
    "    return model, losses\n",
    "    \n",
    "cbow_model, cbow_losses = train_cbow()\n",
    "sg_model, sg_losses = train_skipgram()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====Test CBOW===\n",
      "Accuracy: 94.8% (55/58)\n",
      "------\n",
      "====Test SkipGram===\n",
      "Accuracy: 50.0% (232/464)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rayest/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:12: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  if sys.path[0] == '':\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "# You have to use other dataset for test, but in this case I use training data because this dataset is too small\n",
    "def test_cbow(test_data, model):\n",
    "    print('====Test CBOW===')\n",
    "    correct_ct = 0\n",
    "    for ctx, target in test_data:\n",
    "        ctx_idxs = [w2i[w] for w in ctx]\n",
    "        ctx_var = Variable(torch.LongTensor(ctx_idxs))\n",
    "\n",
    "        model.zero_grad()\n",
    "        log_probs = model(ctx_var)\n",
    "        _, predicted = torch.max(log_probs.data, 1)\n",
    "        #print(i2w)\n",
    "        predicted_word = i2w[int(predicted[0].numpy())]\n",
    "        #print('predicted:', predicted_word)\n",
    "        #print('label    :', target)\n",
    "        if predicted_word == target:\n",
    "            correct_ct += 1\n",
    "            \n",
    "    print('Accuracy: {:.1f}% ({:d}/{:d})'.format(correct_ct/len(test_data)*100, correct_ct, len(test_data)))\n",
    "\n",
    "def test_skipgram(test_data, model):\n",
    "    print('====Test SkipGram===')\n",
    "    correct_ct = 0\n",
    "    for in_w, out_w, target in test_data:\n",
    "        in_w_var = Variable(torch.LongTensor([w2i[in_w]]))\n",
    "        out_w_var = Variable(torch.LongTensor([w2i[out_w]]))\n",
    "\n",
    "        model.zero_grad()\n",
    "        log_probs = model(in_w_var, out_w_var)\n",
    "        _, predicted = torch.max(log_probs.data, 1)\n",
    "        predicted = predicted[0]\n",
    "        if predicted == target:\n",
    "            correct_ct += 1\n",
    "\n",
    "    print('Accuracy: {:.1f}% ({:d}/{:d})'.format(correct_ct/len(test_data)*100, correct_ct, len(test_data)))\n",
    "\n",
    "test_cbow(cbow_train, cbow_model)\n",
    "print('------')\n",
    "test_skipgram(skipgram_train, sg_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3RVVd7G8e8vIfQuoVfpVUpAFEhQaREUK2JBbDRB6lgY553i+87o6BiKBURFLAiioGChyWAK1QDSBUIRQg0gKKDU/f6Ry5jBBJKQcO69eT5rZeVmn3Nzf2edlYfNOfvsbc45REQk+IR4XYCIiOQOBbyISJBSwIuIBCkFvIhIkFLAi4gEqXxeFwBQpkwZV716da/LEBEJKCtWrDjonAvPaLtfBHz16tVJTEz0ugwRkYBiZj9cbLsu0YiIBCkFvIhIkFLAi4gEKQW8iEiQUsCLiAQpBbyISJBSwIuIBKmADvjDx0/x3OcbOH7yjNeliIj4nYAO+ISkg0xavJ1bXklg/Z6jXpcjIuJXAjrgb72mIpMfa83xU2e4/bXFvLt4B1rAREQkVUAHPMB1Na9i9pBI2tYuw19mraff+ys4cuKU12WJiHgu4AMeoHSR/LzdO4I/da3Pwk0H6Do2gcQdh70uS0TEU5cMeDOrYmYLzWyjma03syG+9pfM7HszW2Nmn5pZyTTvGWlmSWa2ycw65+YBpPlMHmt3NdMHXE9oiHHPhKW8tjCJs+d0yUZE8qbM9ODPACOcc/WB1sBAM2sAzAcaOeeaAJuBkQC+bT2BhkAX4HUzC82N4tPTpHJJvhzclpsbV+CluZt4cOIyDvz865X6eBERv3HJgHfO7XXOrfS9/hnYCFRyzs1zzp0fn7gUqOx73R2Y6pw76ZzbDiQBrXK+9IwVKxjG2J5N+eedjVnxw4/cPCae2M0pV7IEERHPZekavJlVB5oByy7Y9Agw2/e6ErArzbZkX9uFv6uvmSWaWWJKSs6Hr5lxT8uqfD6oLVcVKUDvict5/quNnDpzLsc/S0TEH2U64M2sKDAdGOqc+ylN+7OkXsaZfL4pnbf/7kK4c26Ccy7CORcRHp7hgiSXrXa5Yswc1Ib7rq3KG3HbuHPcYramHMu1zxMR8ReZCngzCyM13Cc752akae8NdAPud78NQE8GqqR5e2VgT86Umz0Fw0L5x+2NeaNXC3b9eIJuYxOYunynxsyLSFDLzCgaA94GNjrnYtK0dwGeBm51zp1I85ZZQE8zK2BmNYDawPKcLTt7Ojcsz5whkTSrWpJnZqzl8ckrNWZeRIJWZnrwbYBewI1m9p3v62bgVaAYMN/XNh7AObcemAZsAOYAA51zZ3On/KwrX6IgHzx6LSOj6zF/w36ix8SzZOshr8sSEclx5g+XKSIiIpwXi26vTT7K4Kmr2HHoOAOiajKsYx3CQoPi2S8RyQPMbIVzLiKj7Xk6zRpXLsEXT7SlR4sqvP7NVu4at5gdB497XZaISI7I0wEPUKRAPv55VxPG3d+cHYdO0HVsPB8n7tINWBEJeHk+4M+LblyB2UPa0ahSCZ78ZA0DPljJ4eO6ASsigUsBn0bFkoX4sE9rnomux7+/P0CnUXEs2Ljf67JERLJFAX+B0BCjf1RNZg5qQ5mi+Xn03USemb6GY1o1SkQCjAI+A/UrFGfmoDYMaF+TaYm7iB4Tx/LtmoJYRAKHAv4iCuQL5eku9ZjW7zoM454JS3h+9kZOnvGbYf0iIhlSwGdCRPXSzB7Sjp4tq/JG7Da6v7qIDXt+uvQbRUQ8pIDPpCIF8vH8HY2Z+FAEB4+dovtrCbz+jRYUERH/pYDPohvrlWPesEg6NijHi3M20eONJXo4SkT8kgI+G0oXyc9r9zVn9D1N2bL/Z6LHxPPekh2cU29eRPyIAj6bzIzbmlVi3rAoWtUozZ9nrqfXxGXsPvKL16WJiAAK+MtWvkRBJj3ckn/c3phVO4/QZVQc0zTVgYj4AQV8DjAz7ru2KnOGRFK/YnGe+mQNj72byIGftNi3iHhHAZ+Dql5VmKl9WvM/3RqQkHSQTqPj+GKNp4tZiUgepoDPYSEhxqNta/Dl4HZUu6oIgz5cxaAPV/KjJi4TkStMAZ9LapUtyvT+1/Fk57rMXb+PjqPimLt+n9dliUgeooDPRflCQxh4Qy1mDmxLeLEC9Ht/BYOnrNI0xCJyRSjgr4AGFYsza1AbhnWow+x1e+k0KpY56/Z6XZaIBDkF/BUSFhrCkA61mTWoLeVLFKT/BysZ9OFKDh076XVpIhKkFPBXWP0Kxfn08Tb8oVMd5q7fR6dRcXy5Rr15Ecl5CngPhIWGMOjG2nzxRDsqlSrEwA9X8vjkFRxUb15EcpAC3kN1yxdjxoDreapLXb7ecICOMbHMWr1HT8GKSI5QwHssX2gIj7evxZeD21L1qiIMnrKKfu+vYL+eghWRy6SA9xO1yxVjev/rGBldj9jNKXSIiWXat5rTRkSyTwHvR/KFhtAvqiZzhkZSv0Jxnpq+hl5vL2fX4RNelyYiAUgB74dqlCnC1D6t+b/bGvHdriN0GhXHxITtWj1KRLJEAe+nQkKMB1pXY96wSFpfXZrnvtjA3eMXs2X/z16XJiIBQgHv5yqWLMTEh1oy+p6mbD94nK5jE3hlwRZOnz3ndWki4ucuGfBmVsXMFprZRjNbb2ZDfO2lzWy+mW3xfS/lazczG2tmSWa2xsya5/ZBBLvzq0fNHx5Fp4bleHn+Zm55JYG1yUe9Lk1E/FhmevBngBHOufpAa2CgmTUAngEWOOdqAwt8PwNEA7V9X32BcTledR5VpmgBXr2vORN6teDw8VN0fy2Bf3y1kV9OnfW6NBHxQ5cMeOfcXufcSt/rn4GNQCWgO/Cub7d3gdt8r7sD77lUS4GSZlYhxyvPwzo1LM/84VHc07IKE+K20Xl0HAlbDnpdloj4mSxdgzez6kAzYBlQzjm3F1L/EQDK+narBOxK87ZkX5vkoBKFwnj+jiZM6dOa0BDjgbeX8YePV3PkhKYiFpFUmQ54MysKTAeGOud+utiu6bT9bnyfmfU1s0QzS0xJSclsGXKB62pexewh7Xi8fU0+XbWbDjGxfK7pDkSETAa8mYWRGu6TnXMzfM37z1968X0/4GtPBqqkeXtl4HcLkzrnJjjnIpxzEeHh4dmtX4CCYaE81aUenw9qS8WShXhiyioeezeRPUd+8bo0EfFQZkbRGPA2sNE5F5Nm0yygt+91b2BmmvYHfaNpWgNHz1/KkdzVoGJxZgy4nj91rc+irQfpGBPLe0t2cE4PSInkSXap/8qbWVsgHlgLnB98/UdSr8NPA6oCO4G7nXOHff8gvAp0AU4ADzvnEi/2GRERES4x8aK7SBbtPHSCZz9bS/yWg7SoVooX7mhM7XLFvC5LRHKQma1wzkVkuN0frtUq4HOHc44ZK3fzv19u4PjJMwxoX4uBN9SkQL5Qr0sTkRxwqYDXk6xBzMy4s0Vlvh4eRdfGFRi7YAvRY+JZvv2w16WJyBWggM8DyhQtwOiezZj0cEtOnTlHjzeWMHLGWo7+ctrr0kQkFyng85D2dcsyb1gkfSOv5qNvd9IhJpYv1+zVkEqRIKWAz2MK58/HH2+uz6xBbSlXvAADP1xJn/c0pFIkGCng86hGlUrw2eNtUodUJh2iY0wskxZpznmRYKKAz8PyhYbwWLurmTcskojqpfnr5xu4Y9xiNu692IPKIhIoFPBCldKFmfRwS8b0bEry4RN0eyWB52drlkqRQKeAFyB1SGX3ppVYMCKKu5pX5o3YbXQaHUvsZs0TJBKoFPDyX0oWzs8/72rCR31bExYaQu+Jyxk8ZRUpP5/0ujQRySIFvKTr2qtTZ6kc2qE2c9bt46aXv2HK8p2a10YkgCjgJUMF8oUytEMdZg9tR/0KxRk5Yy33TFhC0gEt/C0SCBTwckk1w4sytW9rXryrCZv3HyN6TDwx8zbx62ndhBXxZwp4yRQzo0dEFRaM8M1r8+8kosfEa6lAET+mgJcsOT+vzfuPtsI5xwNvL2PIVN2EFfFHCnjJlna1w5kzNJLBN9Vm9tp93PjyN3yw9AfdhBXxIwp4ybaCYaEM71iHr4a0o1HFEvzps3XcOX4xG/boSVgRf6CAl8tWq2xRPuxzLTE9rmHnoRPc8moCf/ctMiIi3lHAS44wM+5oXpkFI6LoEVGZN+O30zEmlnnr93ldmkiepYCXHFWycH6ev6MJn/S/jmIFw+j7/goeezeR5B9PeF2aSJ6jgJdcEVG9NF8Mbssz0fVYlHSQjjFxjI/dyumz5y79ZhHJEQp4yTVhoSH0j6rJ/OGRtKlVhhdmf0/XsVoTVuRKUcBLrqtcqjBv9Y7gzQcjOH7yLD3eWMIfPl7NoWMaOy+SmxTwcsV0bFCO+cMj6R9Vk89W7eammFimagIzkVyjgJcrqnD+fDwTXY+vhrSjTtliPDNjLXeN1ypSIrlBAS+eqFOuGB/1a81LdzVhx6HUVaT+74sNHNPYeZEco4AXz5gZd0dUYcHw1LHzbyWkjp2fs24fzumyjcjlUsCL50oVSR07P33AdZQoFEb/D1bw6LuJ7DqssfMil0MBL36jRbXSfPFEW/7UtT5Ltx2i46hYXluYxKkzGjsvkh0KePEr+UJDeKzd1Xw9PIr2dcry0txN3Dw2nqXbDnldmkjAUcCLX6pYshDje7Vg4kMR/Hr6LD0nLGXENI2dF8mKSwa8mU00swNmti5NW1MzW2pm35lZopm18rWbmY01syQzW2NmzXOzeAl+N9Yrx/xhUTzeviazVu/mxpdjtfi3SCZlpgc/CehyQduLwN+cc02BP/t+BogGavu++gLjcqZMycsK5Q/lqS71+GpwO+qWL8ZI39h5zTsvcnGXDHjnXBxw4eQhDijue10C2ON73R14z6VaCpQ0swo5VazkbbXLFeOjvq15+e5r+ME377zGzotkLLvX4IcCL5nZLuBfwEhfeyVgV5r9kn1tv2NmfX2XdxJTUlKyWYbkNWbGnS3OzztfhbcSttPh5Vhmr92rsfMiF8huwA8AhjnnqgDDgLd97ZbOvun+1TnnJjjnIpxzEeHh4dksQ/Kq1HnnGzN9wPWUKpKfAZNX8sikbzV2XiSN7AZ8b2CG7/XHQCvf62SgSpr9KvPb5RuRHNeiWik+H9SGP3Wtz/Lth+kQo7HzIudlN+D3AFG+1zcCW3yvZwEP+kbTtAaOOuf2XmaNIhf1n7HzI6K4qX7q2PnoMXEs2aqx85K3ZWaY5BRgCVDXzJLN7FGgD/Cyma0G/kHqiBmAr4BtQBLwJvB4rlQtko4KJQrx+v0teOfhlpw6e45731zKsI++I+VnjZ2XvMn84cZURESES0xM9LoMCSK/nj7LawuTGB+7lUJhoTwdXY97W1YlJCS920QigcnMVjjnIjLaridZJSgVDAtlRKe6zB4SScOKJXj203XcOX4x6/cc9bo0kStGAS9BrVbZonzY51pG3XMNOw+d4JZXEvhfjZ2XPEIBL0HPzLi9WWX+PaI997aqysRFqWPn56zT2HkJbgp4yTNKFA7j77f/Nna+/wcrNe+8BDUFvOQ5zaumjp3/n24NWKZ55yWIKeAlT8oXGsKjbWvw9Ygobqj727zzy7dfOO2SSOBSwEueVqFEIcY90IJ3HmrJr6fP0uONJTz9yRp+PH7K69JELpsCXgS4oV5Z5g+Lon9UTT5ZmcxNMbHMWJmsm7AS0BTwIj6F8ofyTHQ9vniiLdWuKszwaau5/61lbEs55nVpItmigBe5QP0KxZne/3r+fnsj1u4+SpfR8Yz5egsnz5z1ujSRLFHAi6QjJMS4/9pqLBgRRedG5Rn19Waix8RrAjMJKAp4kYsoW6wgr9zbjHcfacVp3wRmI6at5rBuwkoAUMCLZEJUnXDmDU1d/Hvmd7u56eVv+Dhxl27Cil9TwItk0n8W/x7SjprhRXnykzXc++ZStuomrPgpBbxIFtUpV4xp/a7j+Tsas2HPT0SPjmfU/M38elo3YcW/KOBFsiEkxLi3VVUWjGhPdOPyjFmwhZvHxLN460GvSxP5DwW8yGUIL1aAMT2b8d4jrThzznHfm8t0E1b8hgJeJAdE1gln3rBIBt6gm7DiPxTwIjmkYFgoT3bWTVjxHwp4kRz2u5uwY+J5ZcEWTUcsV5wCXiQXnL8J+/WIKDo1KMfL8zfTdWw8iTs0HbFcOQp4kVxUtlhBXr2vOe881JITp85y1/glPPvpWo7+ctrr0iQPUMCLXAE31CvLvGGRPNa2BlOW76RjTCyz12pNWMldCniRK6RIgXz8qVsDZg5sS3ixAgyYvJI+7yWy58gvXpcmQUoBL3KFNa5cgpkD2/DszfVZlHSIjjGxvLNoO2fPqTcvOUsBL+KBfKEh9Im8mnnDIomoXpq/fb6BO8YtZuPen7wuTYKIAl7EQ1VKF2bSwy0Z07MpyYdPcMsrCbw453vNayM5QgEv4jEzo3vTSiwYEcXtzSrx+jdb6TI6jsVJmtdGLo8CXsRPlCycn5fuvoYPH7sWB9z31jKe/Hg1R05oXhvJnksGvJlNNLMDZrbugvYnzGyTma03sxfTtI80syTfts65UbRIMLu+VhnmDo1kQPuazFi1mw4xscxavUdDKiXLMtODnwR0SdtgZjcA3YEmzrmGwL987Q2AnkBD33teN7PQnCxYJC8oGBbK013q8fmgtlQqWYjBU1bxyKRv2a0hlZIFlwx451wccOHz1QOAF5xzJ337HPC1dwemOudOOue2A0lAqxysVyRPaVCxODMeb8OfuzVg2fbDdIyJZWKChlRK5mT3GnwdoJ2ZLTOzWDNr6WuvBOxKs1+yr+13zKyvmSWaWWJKSko2yxAJfqEhxiNtazBvWCTX1ijNc1+kDqn8fp+GVMrFZTfg8wGlgNbAk8A0MzPA0tk33a6Gc26Ccy7CORcRHh6ezTJE8o7KpQoz8aHfhlR2G5vAv+Zu0pBKyVB2Az4ZmOFSLQfOAWV87VXS7FcZ2HN5JYrIeeeHVH49PIruTSvx6sIkbh4Tz7Jth7wuTfxQdgP+M+BGADOrA+QHDgKzgJ5mVsDMagC1geU5UaiI/KZUkfy83OMa3n+0FafPneOeCUsZOUOzVMp/y8wwySnAEqCumSWb2aPAROBq39DJqUBvX29+PTAN2ADMAQY65/T/R5Fc0q52OHOHRtKnXQ0++jZ1lso56/Z5XZb4CfOHsbUREREuMTHR6zJEAtqa5CM8PX0tG/f+RJeG5flb94aUK17Q67IkF5nZCudcREbb9SSrSJBoUrkkswa14eku9Vi46QAdYmKZsnynHpDKwxTwIkEkLDSEAe1rMmdoJA0rFmfkjLXc9+Yydhw87nVp4gEFvEgQqlGmCFP6tOb5OxqzbvdRuoyJ4824bZw5q4W/8xIFvEiQMktd+Hv+8Cja1grn719t5E49IJWnKOBFglz5EgV588EWvHpfM5J//IVuYxOImbeJk2c0wC3YKeBF8gAzo1uTinw9PIpbr6nI2H8n0XVsAit++NHr0iQXKeBF8pBSRfITc09T3nm4JSdOnuGu8Yv52+frOXHqjNelSS5QwIvkQTfULcu84VH0al2NdxbtoNOoOOK3aNK/YKOAF8mjihbIx3PdGzGt33XkDw2h19vLeeqT1Rw9oekOgoUCXiSPa1WjNF8NaceA9jWZvnI3HUZpuoNgoYAXkf+sIDVzYBvCixag/wcrGDh5JSk/n/S6NLkMCngR+Y9GlUowc1Abnuxcl/kb99MhJpbpK5I13UGAUsCLyH8JCw1h4A21+GpwO2qVLcqIj1fT+51vSf7xhNelSRYp4EUkXbXKFuXjftfxt1sbkrjjMJ1HxfHekh2c03qwAUMBLyIZCgkxel9fnblDI2lerRR/nrmeeyYsYWvKMa9Lk0xQwIvIJVUpXZj3HmnFv+6+hs37jxE9Jp5x32zV5GV+TgEvIpliZtzVojLzh0dyY92y/HPO99z2+iI27NHkZf5KAS8iWVK2WEHG92rBuPubs+/oSW59NYF/zdXkZf5IAS8i2RLduAJfD4/k1qYVeXWhJi/zRwp4Ecm2koXzE9NDk5f5KwW8iFy2Cycv6zw6jkVJB70uK89TwItIjkg7eVm+kBDuf2sZI2es4adfNXmZVxTwIpKjWtUozewh7egXeTUffbuLzqPiWLjpgNdl5UkKeBHJcQXDQhl5c31mPN6GogXy8fA73zJ82nccOXHK69LyFAW8iOSaplVK8sXgtjxxYy1mfreHjqPimLteUxFfKQp4EclVBfKFMqJTXWYObEOZogXo9/4KnpiyikPHNBVxblPAi8gV0ahSCWYNasOIjnWYs24vnUbF8cWaPZqKOBcp4EXkigkLDeGJm2rzxRPtqFyqEIM+XEX/D1Zw4OdfvS4tKCngReSKq1u+GNMHXM8z0fVYuCmFjjFxfLpKC4vktEsGvJlNNLMDZrYunW1/MDNnZmV8P5uZjTWzJDNbY2bNc6NoEQl8+UJD6B9Vk9lDUhcWGfbRah57N5F9R9WbzymZ6cFPArpc2GhmVYCOwM40zdFAbd9XX2Dc5ZcoIsGsZnhRpvW7jj91rc+irQfpOCqWaYm71JvPAZcMeOdcHHA4nU2jgKeAtGehO/CeS7UUKGlmFXKkUhEJWqEhxmPtrmb2kEjqly/OU5+s4aF3vmXPkV+8Li2gZesavJndCux2zq2+YFMlYFean5N9bSIil1SjTBGm9m3NX29pwPLth+k0Ko4py3eqN59NWQ54MysMPAv8Ob3N6bSle2bMrK+ZJZpZYkpKSlbLEJEgFRJiPNSmBnOHRtK4UglGzlhLr7eXs+uwFv3Oquz04GsCNYDVZrYDqAysNLPypPbYq6TZtzKwJ71f4pyb4JyLcM5FhIeHZ6MMEQlmVa8qzOTHruX/bmvEqp0/0mV0HO8v0aLfWZHlgHfOrXXOlXXOVXfOVSc11Js75/YBs4AHfaNpWgNHnXN7c7ZkEckrQkKMB1pXY+6wSJpVLcX/zFzPfW8t5YdDx70uLSBkZpjkFGAJUNfMks3s0Yvs/hWwDUgC3gQez5EqRSRPq1yqMO8/2ooX7mjM+t0/0Xl0HG8nbOesevMXZf5w8yIiIsIlJiZ6XYaIBIC9R3/hjzPWsnBTCi2qleKfdzahVtmiXpflCTNb4ZyLyGi7nmQVkYBSoUQhJj7Ukpge15B04Bg3j41n3DdbOXP2nNel+R0FvIgEHDPjjuaVmT88khvqhvPPOd9zx7jFbNr3s9el+RUFvIgErLLFCjL+gRa8el8zdv/4C91eiWfsgi2cVm8eUMCLSIAzM7o1qci8YZFEN6pAzPzN3PrqItbtPup1aZ5TwItIULiqaAHG3tuMCb1acPDYSW57bREx8zZx6kze7c0r4EUkqHRqWJ75wyK5tWlFxv47iVteSWBN8hGvy/KEAl5Egk7JwvmJ6dGUiQ9FcOSXU9z++mJemvs9J8+c9bq0K0oBLyJB68Z65Zg3LIo7mlXitYVb6TY2gdW78k5vXgEvIkGtRKEwXrr7Gt55uCXHTp7h9tcX8cLs7/n1dPD35hXwIpIn3FC3LHOHRdIjogrjY7fSdWw8K3f+6HVZuUoBLyJ5RvGCYbxwZxPee6QVv5w6y13jFvOPrzYGbW9eAS8ieU5knXDmDoukZ6uqTIjbxs1j4knckd7CdYFNAS8ieVKxgmH84/bGTH7sWk6dPcfdbyzhuc83cOLUGa9LyzEKeBHJ09rUKsPcoZH0al2NiYu2Ez0mnqXbDnldVo5QwItInlekQD6e696IKX1a4xz0nLCUP89cx/GTgd2bV8CLiPhcV/Mq5gxtx8NtqvP+0h/oPDqOxUkHvS4r2xTwIiJpFM6fj7/c0pBp/a4jLDSE+95axh8/XcvPv572urQsU8CLiKSjZfXSfDW4HX3a1WDK8p10GR1P3OYUr8vKEgW8iEgGCuUP5dmuDZg+4HoKhoXw4MTlPPXJao7+Ehi9eQW8iMglNK9aii8Ht2NA+5p8siKZTqNiWbBxv9dlXZICXkQkEwqGhfJ0l3p8NrANJQvl59F3Exn20Xf8ePyU16VlSAEvIpIFTSqX5PMn2jL4ptp8vnoPHUfFMnvtXq/LSpcCXkQki/LnC2F4xzrMGtSWcsULMmDySh6fvIKDx056Xdp/UcCLiGRTg4rF+WxgG57sXJevNxygY0wsM7/bjXPO69IABbyIyGUJCw1h4A21+GJwW6peVYQhU7+jz3sr2P/Tr16XpoAXEckJdcoVY8aA6/njzfWI35JCh5hYPvp2p6e9eQW8iEgOCQ0x+kbWZM7QSOqXL87T09fS6+3l7Dp8wpN6FPAiIjmsRpkiTO3bmv+9rRGrdv5I59FxTFq0nXPnrmxvXgEvIpILQkKMXq2rMW94FC2rl+avn2+gxxtL2Jpy7MrVcMU+SUQkD6pUshCTHm7Jy3dfw5YDx4geE8/r3yRx5uy5XP/sSwa8mU00swNmti5N20tm9r2ZrTGzT82sZJptI80sycw2mVnn3CpcRCRQmBl3tqjM/OGR3FSvLC/O2cRtry9iw56fcvVzM9ODnwR0uaBtPtDIOdcE2AyMBDCzBkBPoKHvPa+bWWiOVSsiEsDKFivIuAdaMO7+5uw7epJbX03g7YTtufZ5lwx451wccPiCtnnOufNLnSwFKvtedwemOudOOue2A0lAqxysV0Qk4EU3rsDXwyO5tWlFql9VONc+J18O/I5HgI98ryuRGvjnJfvafsfM+gJ9AapWrZoDZYiIBI6ShfMT06Nprn7GZd1kNbNngTPA5PNN6eyW7rgg59wE51yEcy4iPDz8csoQEZF0ZLsHb2a9gW7ATe63R7WSgSppdqsM7Ml+eSIikl3Z6sGbWRfgaeBW51zaR7RmAT3NrICZ1QBqA8svv0wREcmqS/bgzWwK0B4oY2bJwF9IHTVTAJhvZgBLnXP9nXPrzWwasIHUSzcDnXNnc6t4ERHJmPnDtJYREREuMTHR6zJERAKKma1wzkVktF1PsoqIBCkFvIhIkFLAi4gEKb+4Bm9mKcAP2Xx7GeBgDpbjD4LtmILteCD4jinYjgeC75jSO55qzrkMHxWsM+QAAAOhSURBVCTyi4C/HGaWeLGbDIEo2I4p2I4Hgu+Ygu14IPiOKTvHo0s0IiJBSgEvIhKkgiHgJ3hdQC4ItmMKtuOB4DumYDseCL5jyvLxBPw1eBERSV8w9OBFRCQdCngRkSAV0AFvZl18a78mmdkzXteTE8xsh5mtNbPvzCzgJujJYA3f0mY238y2+L6X8rLGrMrgmP5qZrt95+k7M7vZyxqzwsyqmNlCM9toZuvNbIivPSDP00WOJ5DPUUEzW25mq33H9Ddfew0zW+Y7Rx+ZWf6L/p5AvQbvW+t1M9CR1HnovwXudc5t8LSwy2RmO4AI51xAPqBhZpHAMeA951wjX9uLwGHn3Au+f4hLOeee9rLOrMjgmP4KHHPO/cvL2rLDzCoAFZxzK82sGLACuA14iAA8Txc5nh4E7jkyoIhz7piZhQEJwBBgODDDOTfVzMYDq51z4zL6PYHcg28FJDnntjnnTgFTSV0TVjyU3hq+pJ6Xd32v3yX1jy9gZHBMAcs5t9c5t9L3+mdgI6lLawbkebrI8QQsl+qY78cw35cDbgQ+8bVf8hwFcsBXAnal+TnD9V8DjAPmmdkK37q1waCcc24vpP4xAmU9rienDDKzNb5LOAFxOeNCZlYdaAYsIwjO0wXHAwF8jsws1My+Aw4A84GtwBHn3BnfLpfMvEAO+Eyv/xpg2jjnmgPRwEDf5QHxP+OAmkBTYC/wsrflZJ2ZFQWmA0Odcz95Xc/lSud4AvocOefOOueakrr0aSugfnq7Xex3BHLAB+X6r865Pb7vB4BPST2xgW6/7zrp+eulBzyu57I55/b7/gDPAW8SYOfJd113OjDZOTfD1xyw5ym94wn0c3Sec+4I8A3QGihpZudX4rtk5gVywH8L1PbdVc4P9CR1TdiAZWZFfDeJMLMiQCdg3cXfFRBmAb19r3sDMz2sJUecD0Kf2wmg8+S7gfc2sNE5F5NmU0Cep4yOJ8DPUbiZlfS9LgR0IPXewkLgLt9ulzxHATuKBsA37Gk0EApMdM793eOSLouZXU1qrx1S18v9MNCOKe0avsB+Utfw/QyYBlQFdgJ3O+cC5qZlBsfUntT/+jtgB9Dv/PVrf2dmbYF4YC1wztf8R1KvWwfcebrI8dxL4J6jJqTeRA0ltSM+zTn3nC8jpgKlgVXAA865kxn+nkAOeBERyVggX6IREZGLUMCLiAQpBbyISJBSwIuIBCkFvIhIkFLAi4gEKQW8iEiQ+n/rQE5Zy1VoegAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAdt0lEQVR4nO3de5Bc5X3m8e/Tc5GmBZqW0IB1IxK24jUGEcgEsB2nXGCDMGCRWpOCbBbFS0qJjTfOOlsx3lQtWTts2VlvSKjYeGWjtUixYBY7i8DYWME4Xq+5jTCIi4w1BgyDhDRYF0DXufz2j36n1TPTc+seqWfmPJ+qqTnnPe85/R63mUfnfc95jyICMzMzgFy9G2BmZlOHQ8HMzEocCmZmVuJQMDOzEoeCmZmVNNa7AdVasGBBLFu2rN7NMDObVjZv3vx6RLSNtH3ahsKyZcvo6OiodzPMzKYVSb8cbbu7j8zMrMShYGZmJQ4FMzMrcSiYmVmJQ8HMzEocCmZmVuJQMDOzksyFwoafvMTGp7bXuxlmZlNS5kLhjsde5l6HgplZRZkLhdaWJvYd7Kl3M8zMpqTMhUIh38S+Aw4FM7NKshcKLc3sPXik3s0wM5uSshcK+Sb2+krBzKyizIVCa76Jw739HOrpq3dTzMymnMyFQqGlGcBXC2ZmFWQvFPJNAB5XMDOrIHuh0JJCwVcKZmbDjBkKktZL2iXpmQrb/qOkkLQgrUvSzZI6JW2RdE5Z3TWStqWfNWXlvynp6bTPzZI0WSdXSWveoWBmNpLxXCl8A1g1tFDSUuBDwMtlxZcAK9LPWuCWVHc+cANwHnAucIOkeWmfW1Ldgf2GfdZkKuSLYwr73H1kZjbMmKEQET8CdlfYdBPwF0CUla0GbouiR4CCpIXAxcCmiNgdEXuATcCqtG1uRDwcEQHcBlxR2ymNzt1HZmYjq2pMQdJHgFcj4qkhmxYDr5Std6Wy0cq7KpSP9LlrJXVI6uju7q6m6eSbG2hqEHs91YWZ2TATDgVJeeAvgf9caXOFsqiivKKIWBcR7RHR3tbWNp7mDm+gRGtLs68UzMwqqOZK4e3AcuApSS8BS4AnJL2N4r/0l5bVXQJsH6N8SYXyY6qQb/KYgplZBRMOhYh4OiJOjohlEbGM4h/2cyLiNWAjcE26C+l8YF9E7AAeAC6SNC8NMF8EPJC2vSnp/HTX0TXAPZN0biMqtHiqCzOzSsZzS+odwMPAOyV1Sbp2lOr3Ay8AncDXgE8ARMRu4PPA4+nnc6kM4OPA19M+vwC+W92pjF+rQ8HMrKLGsSpExNVjbF9WthzAdSPUWw+sr1DeAZwxVjsmU2u+iZ+99ubx/Egzs2khc080Q3H+I79ox8xsuGyGQr6Jtw730tPXX++mmJlNKZkNBcBXC2ZmQ2QyFFr9VLOZWUWZDAXPf2RmVlk2Q8FXCmZmFWUzFDx9tplZRdkMhYFXcnqg2cxskEyGwomzG5Fg3wGPKZiZlctkKORyKk514SsFM7NBMhkK4EnxzMwqyWwotOabfaVgZjZEdkOhpcljCmZmQ2Q2FAoeUzAzGya7oZBv8txHZmZDZDcUWoqh0N8/4iuhzcwyJ7Oh0JpvJgLePNRb76aYmU0ZmQ2F0vxHnhTPzKxkPO9oXi9pl6Rnysr+m6SfSdoi6Z8kFcq2fVZSp6TnJV1cVr4qlXVKur6sfLmkRyVtk/RNSc2TeYIj8fxHZmbDjedK4RvAqiFlm4AzImIl8HPgswCSTgeuAt6d9vmKpAZJDcCXgUuA04GrU12ALwI3RcQKYA9wbU1nNE6lUPBgs5lZyZihEBE/AnYPKft+RAx0xj8CLEnLq4E7I+JwRLwIdALnpp/OiHghIo4AdwKrJQm4ALg77b8BuKLGcxqX1oFJ8fysgplZyWSMKfw74LtpeTHwStm2rlQ2UvlJwN6ygBkor0jSWkkdkjq6u7trarRfyWlmNlxNoSDpL4Fe4PaBogrVooryiiJiXUS0R0R7W1vbRJs7iF/JaWY2XGO1O0paA1wGXBgRA3/Iu4ClZdWWANvTcqXy14GCpMZ0tVBe/5hqashxwqxGh4KZWZmqrhQkrQI+A3wkIg6UbdoIXCVplqTlwArgMeBxYEW606iZ4mD0xhQmDwEfTfuvAe6p7lQmrjh9tscUzMwGjOeW1DuAh4F3SuqSdC3wD8CJwCZJT0r6KkBEPAvcBTwHfA+4LiL60lXAJ4EHgK3AXakuFMPl05I6KY4x3DqpZziKQr6Jfb5SMDMrGbP7KCKurlA84h/uiLgRuLFC+f3A/RXKX6B4d9JxV8h7Ujwzs3KZfaIZUveRb0k1MyvJeCg0+5ZUM7MymQ6FQr74Ss6jN0+ZmWVbtkOhpYne/uDAkb56N8XMbErIdih4/iMzs0EyHQqe/8jMbLBMh0Jp/iM/q2BmBjgUAHcfmZkNyHYolLqPHApmZpD1UMj7lZxmZuUyHQqzmxqY1ZjzmIKZWZLpUICjD7CZmZlDgUJLs7uPzMySzIdCq68UzMxKMh8KhZYmT4pnZpZkPhSK02c7FMzMwKGQXrTjMQUzM3AoUMg3c6inn0M9ninVzGw872heL2mXpGfKyuZL2iRpW/o9L5VL0s2SOiVtkXRO2T5rUv1tktaUlf+mpKfTPjdL0mSf5GhaW9L8Rx5XMDMb15XCN4BVQ8quBx6MiBXAg2kd4BJgRfpZC9wCxRABbgDOo/g+5hsGgiTVWVu239DPOqZKk+I5FMzMxg6FiPgRsHtI8WpgQ1reAFxRVn5bFD0CFCQtBC4GNkXE7ojYA2wCVqVtcyPi4Si+/uy2smMdF57/yMzsqGrHFE6JiB0A6ffJqXwx8EpZva5UNlp5V4XyiiStldQhqaO7u7vKpg9Wmv/I71QwM5v0geZK4wFRRXlFEbEuItojor2tra3KJg42MKbg6bPNzKoPhZ2p64f0e1cq7wKWltVbAmwfo3xJhfLjxi/aMTM7qtpQ2AgM3EG0BrinrPyadBfS+cC+1L30AHCRpHlpgPki4IG07U1J56e7jq4pO9ZxccKsRhpy8rMKZmZA41gVJN0BfABYIKmL4l1EXwDuknQt8DJwZap+P/BhoBM4AHwMICJ2S/o88Hiq97mIGBi8/jjFO5xagO+mn+NGEgU/1WxmBowjFCLi6hE2XVihbgDXjXCc9cD6CuUdwBljteNYas03eUzBzAw/0QykSfF8pWBm5lCA4lQXHlMwM3MoAJ4p1cxsgEOBYii4+8jMzKEAFJ9VePNwLz19/fVuiplZXTkUKA40A7zhO5DMLOMcChQHmsFTXZiZORQoPqcAnj7bzMyhwNHuIw82m1nWORQo7z7yswpmlm0OBY5eKfhZBTPLOocCMNehYGYGOBQAaMiJubMbPdBsZpnnUEgK+Wa/ktPMMs+hkBQ8fbaZmUNhgCfFMzNzKJQU8s0eUzCzzHMoJK0tjR5TMLPMqykUJP0HSc9KekbSHZJmS1ou6VFJ2yR9U1JzqjsrrXem7cvKjvPZVP68pItrO6XqFFqKVwr9/VGPjzczmxKqDgVJi4E/Bdoj4gygAbgK+CJwU0SsAPYA16ZdrgX2RMQ7gJtSPSSdnvZ7N7AK+IqkhmrbVa1Cvon+gDcP9x7vjzYzmzJq7T5qBFokNQJ5YAdwAXB32r4BuCItr07rpO0XSlIqvzMiDkfEi0AncG6N7ZqwVs9/ZGZWfShExKvAl4CXKYbBPmAzsDciBv653QUsTsuLgVfSvr2p/knl5RX2GUTSWkkdkjq6u7urbXpFnv/IzKy27qN5FP+VvxxYBMwBLqlQdaCTXiNsG6l8eGHEuohoj4j2tra2iTd6FIW8p7owM6ul++iDwIsR0R0RPcC3gfcChdSdBLAE2J6Wu4ClAGl7K7C7vLzCPsdNafps35ZqZhlWSyi8DJwvKZ/GBi4EngMeAj6a6qwB7knLG9M6afsPIiJS+VXp7qTlwArgsRraVZWBF+34qWYzy7LGsatUFhGPSrobeALoBX4KrAO+A9wp6a9T2a1pl1uBf5TUSfEK4ap0nGcl3UUxUHqB6yKir9p2VevoQLPHFMwsu6oOBYCIuAG4YUjxC1S4eygiDgFXjnCcG4Eba2lLrWY1NpBvbvCYgpllmp9oLlNo8aR4ZpZtDoUyrflmXymYWaY5FMoUWprY5+cUzCzDHAplCnlPn21m2eZQKNPqMQUzyziHQpnWfBP7DvRQfHzCzCx7HAplCi3NHOnr52DPcX9MwsxsSnAolPH8R2aWdQ6FMgPzHzkUzCyrHApljs5/5NtSzSybHAplCi3Fdyr4RTtmllUOhTIFz5RqZhnnUCgzEAp+p4KZZZVDoUxLUwPNDTkPNJtZZjkUykgqPsDmgWYzyyiHwhCFFs9/ZGbZ5VAYwpPimVmWORSGaG1p9t1HZpZZNYWCpIKkuyX9TNJWSe+RNF/SJknb0u95qa4k3SypU9IWSeeUHWdNqr9N0ppaT6oWhXyT39NsZplV65XC3wPfi4h/BZwFbAWuBx6MiBXAg2kd4BJgRfpZC9wCIGk+xfc8n0fx3c43DARJPXj6bDPLsqpDQdJc4HeAWwEi4khE7AVWAxtStQ3AFWl5NXBbFD0CFCQtBC4GNkXE7ojYA2wCVlXbrloVWpo4cKSPw72eKdXMsqeWK4XTgG7gf0r6qaSvS5oDnBIROwDS75NT/cXAK2X7d6WykcqHkbRWUoekju7u7hqaPjI/wGZmWVZLKDQC5wC3RMTZwH6OdhVVogplMUr58MKIdRHRHhHtbW1tE23vuLTmPf+RmWVXLaHQBXRFxKNp/W6KIbEzdQuRfu8qq7+0bP8lwPZRyuuiNH22rxTMLIOqDoWIeA14RdI7U9GFwHPARmDgDqI1wD1peSNwTboL6XxgX+peegC4SNK8NMB8USqrC79ox8yyrLHG/f89cLukZuAF4GMUg+YuSdcCLwNXprr3Ax8GOoEDqS4RsVvS54HHU73PRcTuGttVtYHps/f6tlQzy6CaQiEingTaK2y6sELdAK4b4TjrgfW1tGWytHqg2cwyzE80D3HirEZyciiYWTY5FIbI5VR8gM1jCmaWQQ6FCgp5z39kZtnkUKigeKXggWYzyx6HQgWFfJPHFMwskxwKFfhFO2aWVQ6FCtx9ZGZZ5VCooDXfzBuHeunrrzgFk5nZjOVQqGBg/qM3PK5gZhnjUKigNP+RQ8HMMsahUMHRSfE8rmBm2eJQqKB1YFI8XymYWcY4FCoovX3Nt6WaWcY4FCoovWjH3UdmljEOhQpa/fY1M8soh0IFjQ05TpzV6KeazSxzHAojaM03+TkFM8sch8IICvkmdx+ZWebUHAqSGiT9VNJ9aX25pEclbZP0zfT+ZiTNSuudafuysmN8NpU/L+niWts0GQotzR5oNrPMmYwrhU8BW8vWvwjcFBErgD3Atan8WmBPRLwDuCnVQ9LpwFXAu4FVwFckNUxCu2rS6isFM8ugmkJB0hLgUuDraV3ABcDdqcoG4Iq0vDqtk7ZfmOqvBu6MiMMR8SLQCZxbS7smQ6Glyc8pmFnm1Hql8HfAXwD9af0kYG9E9Kb1LmBxWl4MvAKQtu9L9UvlFfYZRNJaSR2SOrq7u2ts+uhaW4pXChGeKdXMsqPqUJB0GbArIjaXF1eoGmNsG22fwYUR6yKiPSLa29raJtTeiSrkm+jrD9463Dt2ZTOzGaKxhn3fB3xE0oeB2cBcilcOBUmN6WpgCbA91e8ClgJdkhqBVmB3WfmA8n3qpjAw/9GBHk6c3VTn1piZHR9VXylExGcjYklELKM4UPyDiPg3wEPAR1O1NcA9aXljWidt/0EU+2Y2Alelu5OWAyuAx6pt12R5+8lzANj8yz11bomZ2fFzLJ5T+AzwaUmdFMcMbk3ltwInpfJPA9cDRMSzwF3Ac8D3gOsiou8YtGtCzl46j7fNnc19W+p+0WJmdtzU0n1UEhE/BH6Yll+gwt1DEXEIuHKE/W8EbpyMtkyWXE5cunIhtz38EvsO9NCadxeSmc18fqJ5FJeftYievuCB516rd1PMzI4Lh8IozlrSytL5Ldy3ZUe9m2Jmdlw4FEYhiUvPXMT/63yd3fs95YWZzXwOhTFcftZC+vqD7z7jqwUzm/kcCmM4feFcTlswh/ueciiY2cznUBiDJC5buZBHXvwVu944VO/mmJkdUw6Fcbj8rEVEwP1P+2rBzGY2h8I4rDjlRN55yom+C8nMZjyHwjhdtnIhHb/cw/a9B+vdFDOzY8ahME6XnbUIgO/4asHMZjCHwjgtXzCHMxbP9VxIZjajORQm4LKVi3iqax8v/+pAvZtiZnZMOBQm4NIzFwJwr68WzGyGcihMwNL5ec4+teC7kMxsxnIoTNBlKxexdccb/KL7rXo3xcxs0jkUJujSMxci4WkvzGxGcihM0NtaZ/Nby+Zz75btFN8mamY2czgUqnD5yoV07nqL53e+We+mmJlNqqpDQdJSSQ9J2irpWUmfSuXzJW2StC39npfKJelmSZ2Stkg6p+xYa1L9bZLW1H5ax9YlZy4k5y4kM5uBarlS6AX+PCLeBZwPXCfpdOB64MGIWAE8mNYBLgFWpJ+1wC1QDBHgBuA8iu92vmEgSKaqBSfM4r1vX+AuJDObcaoOhYjYERFPpOU3ga3AYmA1sCFV2wBckZZXA7dF0SNAQdJC4GJgU0Tsjog9wCZgVbXtOl4uW7mQX/7qAM+8+ka9m2JmNmkmZUxB0jLgbOBR4JSI2AHF4ABOTtUWA6+U7daVykYqr/Q5ayV1SOro7u6ejKZXbdUZb6MxJ097YWYzSs2hIOkE4FvAn0XEaP9sVoWyGKV8eGHEuohoj4j2tra2iTd2EhXyzbx/xQLu27LDXUhmNmPUFAqSmigGwu0R8e1UvDN1C5F+70rlXcDSst2XANtHKZ/yLlu5iFf3HuSJl/fWuylmZpOilruPBNwKbI2Ivy3btBEYuINoDXBPWfk16S6k84F9qXvpAeAiSfPSAPNFqWzK+9C7T6G5MecuJDObMWq5Ungf8G+BCyQ9mX4+DHwB+JCkbcCH0jrA/cALQCfwNeATABGxG/g88Hj6+Vwqm/Lmzm7iA7/exne27KCv311IZjb9NVa7Y0T8mMrjAQAXVqgfwHUjHGs9sL7attTTZWct4vvP7eTxl3Zz/mkn1bs5ZmY18RPNNfrgu06mpamBr/3oBXr6+uvdHDOzmjgUapRvbuTPL/p1HvzZLv74Hzdz8EhfvZtkZlY1h8Ik+KP3n8Z//d0z+eHzu/iDWx9l34GeejfJzKwqDoVJ8vvnncqXf/8cnu7ax+/9j4fZ+cahejfJzGzCHAqT6JIzF/KNj/0WXXsO8K9v+Qkvvr6/3k0yM5sQh8Ike+87FnDH2vM5cKSPK7/6E555dV+9m2RmNm4OhWNg5ZICd//Je5jV2MBV6x7hJ794vd5NMjMbF4fCMXJa2wl86+PvZWHrbP5w/eN875nX6t0kM7MxORSOobe1zuZ//8l7ePfiuXzi9s3c+djL9W6SmdmoHArHWCHfzO1/dB7vX9HG9d9+mn/4wTaO9PohNzObmhwKx0G+uZGvXdPOR85axJe+/3Pa/3oTn7l7Cz/e9jq9fgrazKYQTdd3AbS3t0dHR0e9mzEh/f3Bv/y8m3uf2s73n9vJW4d7WXBCM5eeuZDLz1rEOafOI5cbaTopM7PaSdocEe0jbnco1Mehnj5++Pwu7n1qB/+8dSeHe/tZ1Dqby85axOUrF3HG4rkUZyc3M5s8DoVp4K3Dvfzzczu596nt/MvPu+ntD5YvmMN73n4Sy0+aw/IFc1i2YA6nzs/T3OgePzOr3lihUPXU2TZ5TpjVyBVnL+aKsxez98ARvvfMa9y3ZQff2bKDfQePzqOUEyyZl2f5ghQUJ+VZ3nYCvzY/z7w5zZw4q9HdT2ZWE4fCFFPIN3PVuady1bmnArBn/xFe/NV+Xnp9Py+mn5d+tZ+Ol3azf8iMrDlBa0sThXxz+t1EYcj6nFmNtDQ10NLUQL65gdnNDaX1luYGZqflpga5+8osgxwKU9y8Oc3Mm9PMOafOG1QeEXS/dZgXu/fzyp6D7D1whH0He9h7oIe9B3vYe+AIu/cf4YXu/ew9cIQ3DvVO6HMbcqKpQTQ15JjVmKOpYeBneFljg2jMicaGHI050ZAbvt7UkCMn0dggchINOWiQyOVU+j1Qt7hd5AS5tJ4T6bfI5Y4uN+SEAJXXyYEQKt9HQFoXxeMO7Kfy8rQuHT3GwHIuLcPgfcrrMrBeYVvadXBdhtel/Dhp+9G6qQ4M/ryysor7OOBtnBwK05QkTj5xNiefOJvzxlG/rz/Yd7CH/Yd7OdTTx8GePg4eKf4+ut5/dP1IHz19/Rzp6y/+7u2npy84Ulo+Wn6oN+jtC3r7g77+/rLloLdsvbevn74I+vuhL8KvMK2joaE0sDx4W1mlUbaX583ROoOPOfwYQ4JtSPmgfQd9hiqUDX8F5OBto+8zUmAOLR7pmGO3ZfjxK37i0M8b5Tjf+dPfZlZjQ6Wj1GzKhIKkVcDfAw3A1yPiC2PsYhPQkBPz5zQzf05zvZsySH9/lAKiLy339wf9UQyyiLScymNgOa33RbEsAvrTcv/A9gAoPxYEQ+pT/M2Q/YPi1VjaVNq3P4rlcLT+0TrF/Sj/nLJ9B9ZJ+5e2lT4vbefoZwzcBzJo/yHHpGyf0eoEZRvKPn/g+IM/jyHrgzcMbed49h16T0uUtWPo9mDk4w4/1uADD/7fpLpjBkMbO9KnDfnfYNg2hqn0T6GhN/wMqzOkYGgoTaYpEQqSGoAvAx8CuoDHJW2MiOfq2zI71nI5kUM0HZt/9JjZBE2V+xvPBToj4oWIOALcCayuc5vMzDJnqoTCYuCVsvWuVDaIpLWSOiR1dHd3H7fGmZllxVQJhUodZMO71SLWRUR7RLS3tbUdh2aZmWXLVAmFLmBp2foSYHud2mJmlllTJRQeB1ZIWi6pGbgK2FjnNpmZZc6UuPsoInolfRJ4gOItqesj4tk6N8vMLHOmRCgARMT9wP31boeZWZZNle4jMzObAqbt1NmSuoFfVrn7AuD1SWxOvc2084GZd04z7Xxg5p3TTDsfqHxOvxYRI96+OW1DoRaSOkabT3y6mWnnAzPvnGba+cDMO6eZdj5Q3Tm5+8jMzEocCmZmVpLVUFhX7wZMspl2PjDzzmmmnQ/MvHOaaecDVZxTJscUzMyssqxeKZiZWQUOBTMzK8lUKEhaJel5SZ2Srq93eyaDpJckPS3pSUkd9W5PNSStl7RL0jNlZfMlbZK0Lf2eN9oxppIRzuevJL2avqcnJX24nm2cCElLJT0kaaukZyV9KpVP5+9opHOalt+TpNmSHpP0VDqf/5LKl0t6NH1H30xzy41+rKyMKaS3u/2csre7AVdP97e7SXoJaI+IafvQjaTfAd4CbouIM1LZ3wC7I+ILKcDnRcRn6tnO8RrhfP4KeCsivlTPtlVD0kJgYUQ8IelEYDNwBfCHTN/vaKRz+j2m4fek4guc50TEW5KagB8DnwI+DXw7Iu6U9FXgqYi4ZbRjZelKwW93m6Ii4kfA7iHFq4ENaXkDxf9gp4URzmfaiogdEfFEWn4T2ErxJVjT+Tsa6ZympSh6K602pZ8ALgDuTuXj+o6yFArjervbNBTA9yVtlrS23o2ZRKdExA4o/gcMnFzn9kyGT0rakrqXpk1XSzlJy4CzgUeZId/RkHOCafo9SWqQ9CSwC9gE/ALYGxG9qcq4/uZlKRTG9Xa3aeh9EXEOcAlwXeq6sKnnFuDtwG8AO4D/Xt/mTJykE4BvAX8WEW/Uuz2TocI5TdvvKSL6IuI3KL6k7FzgXZWqjXWcLIXCjHy7W0RsT793Af9E8f8MM8HO1O870P+7q87tqUlE7Ez/0fYDX2OafU+pn/pbwO0R8e1UPK2/o0rnNN2/J4CI2Av8EDgfKEgaeEXCuP7mZSkUZtzb3STNSYNkSJoDXAQ8M/pe08ZGYE1aXgPcU8e21Gzgj2fyu0yj7ykNYt4KbI2Ivy3bNG2/o5HOabp+T5LaJBXScgvwQYrjJA8BH03VxvUdZebuI4B0e9nfcfTtbjfWuUk1kXQaxasDKL4w6X9Nx3OSdAfwAYrT/O4EbgD+D3AXcCrwMnBlREyLwdsRzucDFLskAngJ+OOB/vipTtJvA/8XeBroT8X/iWIf/HT9jkY6p6uZht+TpJUUB5IbKP5j/66I+Fz6G3EnMB/4KfAHEXF41GNlKRTMzGx0Weo+MjOzMTgUzMysxKFgZmYlDgUzMytxKJiZWYlDwczMShwKZmZW8v8BSj4WJerUcQsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def showPlot(points, title):\n",
    "    plt.figure()\n",
    "    fig, ax = plt.subplots()\n",
    "    plt.plot(points)\n",
    "\n",
    "showPlot(cbow_losses, 'CBOW Losses')\n",
    "showPlot(sg_losses, 'SkipGram Losses')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
